{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Image Classification with TensorFlow on Cloud AI Platform\n",
    "\n",
    "This notebook demonstrates how to implement different image models on MNIST using the [tf.keras API](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras).\n",
    "\n",
    "## Learning objectives\n",
    "1. Understand how to build a Dense Neural Network (DNN) for image classification\n",
    "2. Understand how to use dropout (DNN) for image classification\n",
    "3. Understand how to use Convolutional Neural Networks (CNN)\n",
    "4. Know how to deploy and use an image classification model using Google Cloud's [Vertex AI](https://cloud.google.com/vertex-ai/)\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/2_mnist_models.ipynb) -- try to complete that notebook first before reviewing this solution notebook.\n",
    "\n",
    "First things first. Configure the parameters below to match your own Google Cloud project details."
   ]
  },
  {
    "cell_type": "code",
    "metadata": {
      "id": "Nny3m465gKkY",
      "colab_type": "code",
      "colab": {}
    },
    "source": [
      "!sudo chown -R jupyter:jupyter /home/jupyter/training-data-analyst"
    ],
    "execution_count": null,
    "outputs": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "2.6.0\n"
      ]
     }
   ],
   "source": [
    "# Here we'll show the currently installed version of TensorFlow\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "import os\n",
    "\n",
    "PROJECT = \"your-project-id-here\"  # REPLACE WITH YOUR PROJECT ID\n",
    "BUCKET = \"your-bucket-id-here\"  # REPLACE WITH YOUR BUCKET NAME\n",
    "REGION = \"us-central1\"  # REPLACE WITH YOUR BUCKET REGION e.g. us-central1\n",
    "MODEL_TYPE = \"cnn\"  # \"linear\", \"cnn\", \"dnn_dropout\", or \"dnn\"\n",
    "\n",
    "# Do not change these\n",
    "os.environ[\"PROJECT\"] = PROJECT\n",
    "os.environ[\"BUCKET\"] = BUCKET\n",
    "os.environ[\"REGION\"] = REGION\n",
    "os.environ[\"MODEL_TYPE\"] = MODEL_TYPE\n",
    "os.environ[\"TFVERSION\"] = \"2.6\"  # Tensorflow  version\n",
    "os.environ[\"IMAGE_URI\"] = os.path.join(\"gcr.io\", PROJECT, \"mnist_models\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a dynamic model\n",
    "\n",
    "In the previous notebook, <a href=\"1_mnist_linear.ipynb\">1_mnist_linear.ipynb</a>, we ran our code directly from the notebook. In order to run it on the AI Platform, it needs to be packaged as a python module.\n",
    "\n",
    "The boilerplate structure for this module has already been set up in the folder `mnist_models`. The module lives in the sub-folder, `trainer`, and is designated as a python package with the empty `__init__.py` (`mnist_models/trainer/__init__.py`) file. It still needs the model and a trainer to run it, so let's make them.\n",
    "\n",
    "Let's start with the trainer file first. This file parses command line arguments to feed into the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mnist_models/trainer/task.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile mnist_models/trainer/task.py\n",
    "import argparse\n",
    "import json\n",
    "import os\n",
    "import sys\n",
    "\n",
    "from . import model\n",
    "\n",
    "\n",
    "def _parse_arguments(argv):\n",
    "    \"\"\"Parses command-line arguments.\"\"\"\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\n",
    "        '--model_type',\n",
    "        help='Which model type to use',\n",
    "        type=str, default='linear')\n",
    "    parser.add_argument(\n",
    "        '--epochs',\n",
    "        help='The number of epochs to train',\n",
    "        type=int, default=10)\n",
    "    parser.add_argument(\n",
    "        '--steps_per_epoch',\n",
    "        help='The number of steps per epoch to train',\n",
    "        type=int, default=100)\n",
    "    parser.add_argument(\n",
    "        '--job-dir',\n",
    "        help='Directory where to save the given model',\n",
    "        type=str, default='mnist_models/')\n",
    "    return parser.parse_known_args(argv)\n",
    "\n",
    "\n",
    "def main():\n",
    "    \"\"\"Parses command line arguments and kicks off model training.\"\"\"\n",
    "    args = _parse_arguments(sys.argv[1:])[0]\n",
    "\n",
    "    # Configure path for hyperparameter tuning.\n",
    "    trial_id = json.loads(\n",
    "        os.environ.get('TF_CONFIG', '{}')).get('task', {}).get('trial', '')\n",
    "    output_path = args.job_dir if not trial_id else args.job_dir + '/'\n",
    "\n",
    "    model_layers = model.get_layers(args.model_type)\n",
    "    image_model = model.build_model(model_layers, args.job_dir)\n",
    "    model_history = model.train_and_evaluate(\n",
    "        image_model, args.epochs, args.steps_per_epoch, args.job_dir)\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's group non-model functions into a util file to keep the model file simple. We'll copy over the `scale` and `load_dataset` functions from the previous lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mnist_models/trainer/util.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile mnist_models/trainer/util.py\n",
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "def scale(image, label):\n",
    "    \"\"\"Scales images from a 0-255 int range to a 0-1 float range\"\"\"\n",
    "    image = tf.cast(image, tf.float32)\n",
    "    image /= 255\n",
    "    image = tf.expand_dims(image, -1)\n",
    "    return image, label\n",
    "\n",
    "\n",
    "def load_dataset(\n",
    "        data, training=True, buffer_size=5000, batch_size=100, nclasses=10):\n",
    "    \"\"\"Loads MNIST dataset into a tf.data.Dataset\"\"\"\n",
    "    (x_train, y_train), (x_test, y_test) = data\n",
    "    x = x_train if training else x_test\n",
    "    y = y_train if training else y_test\n",
    "    # One-hot encode the classes\n",
    "    y = tf.keras.utils.to_categorical(y, nclasses)\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((x, y))\n",
    "    dataset = dataset.map(scale).batch(batch_size)\n",
    "    if training:\n",
    "        dataset = dataset.shuffle(buffer_size).repeat()\n",
    "    return dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's code the models! The [tf.keras API](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras) accepts an array of [layers](https://www.tensorflow.org/api_docs/python/tf/keras/layers) into a [model object](https://www.tensorflow.org/api_docs/python/tf/keras/Model), so we can create a dictionary of layers based on the different model types we want to use. The below file has two functions: `get_layers` and `create_and_train_model`. We will build the structure of our model in `get_layers`. Last but not least, we'll copy over the training code from the previous lab into `train_and_evaluate`.\n",
    "\n",
    "**TODO 1**: Define the Keras layers for a DNN model   \n",
    "**TODO 2**: Define the Keras layers for a dropout model  \n",
    "**TODO 3**: Define the Keras layers for a CNN model  \n",
    "\n",
    "Hint: These models progressively build on each other. Look at the imported `tensorflow.keras.layers` modules and the default values for the variables defined in `get_layers` for guidance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mnist_models/trainer/model.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile mnist_models/trainer/model.py\n",
    "import os\n",
    "import shutil\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.callbacks import TensorBoard\n",
    "from tensorflow.keras.layers import (\n",
    "    Conv2D, Dense, Dropout, Flatten, MaxPooling2D, Softmax)\n",
    "\n",
    "from . import util\n",
    "\n",
    "\n",
    "# Image Variables\n",
    "WIDTH = 28\n",
    "HEIGHT = 28\n",
    "\n",
    "\n",
    "def get_layers(\n",
    "        model_type,\n",
    "        nclasses=10,\n",
    "        hidden_layer_1_neurons=400,\n",
    "        hidden_layer_2_neurons=100,\n",
    "        dropout_rate=0.25,\n",
    "        num_filters_1=64,\n",
    "        kernel_size_1=3,\n",
    "        pooling_size_1=2,\n",
    "        num_filters_2=32,\n",
    "        kernel_size_2=3,\n",
    "        pooling_size_2=2):\n",
    "    \"\"\"Constructs layers for a keras model based on a dict of model types.\"\"\"\n",
    "    model_layers = {\n",
    "        'linear': [\n",
    "            Flatten(),\n",
    "            Dense(nclasses),\n",
    "            Softmax()\n",
    "        ],\n",
    "        'dnn': [\n",
    "            Flatten(),\n",
    "            Dense(hidden_layer_1_neurons, activation='relu'),\n",
    "            Dense(hidden_layer_2_neurons, activation='relu'),\n",
    "            Dense(nclasses),\n",
    "            Softmax()\n",
    "        ],\n",
    "        'dnn_dropout': [\n",
    "            Flatten(),\n",
    "            Dense(hidden_layer_1_neurons, activation='relu'),\n",
    "            Dense(hidden_layer_2_neurons, activation='relu'),\n",
    "            Dropout(dropout_rate),\n",
    "            Dense(nclasses),\n",
    "            Softmax()\n",
    "        ],\n",
    "        'cnn': [\n",
    "            Conv2D(num_filters_1, kernel_size=kernel_size_1,\n",
    "                   activation='relu', input_shape=(WIDTH, HEIGHT, 1)),\n",
    "            MaxPooling2D(pooling_size_1),\n",
    "            Conv2D(num_filters_2, kernel_size=kernel_size_2,\n",
    "                   activation='relu'),\n",
    "            MaxPooling2D(pooling_size_2),\n",
    "            Flatten(),\n",
    "            Dense(hidden_layer_1_neurons, activation='relu'),\n",
    "            Dense(hidden_layer_2_neurons, activation='relu'),\n",
    "            Dropout(dropout_rate),\n",
    "            Dense(nclasses),\n",
    "            Softmax()\n",
    "        ]\n",
    "    }\n",
    "    return model_layers[model_type]\n",
    "\n",
    "\n",
    "def build_model(layers, output_dir):\n",
    "    \"\"\"Compiles keras model for image classification.\"\"\"\n",
    "    model = Sequential(layers)\n",
    "    model.compile(optimizer='adam',\n",
    "                  loss='categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "\n",
    "def train_and_evaluate(model, num_epochs, steps_per_epoch, output_dir):\n",
    "    \"\"\"Compiles keras model and loads data into it for training.\"\"\"\n",
    "    mnist = tf.keras.datasets.mnist.load_data()\n",
    "    train_data = util.load_dataset(mnist)\n",
    "    validation_data = util.load_dataset(mnist, training=False)\n",
    "\n",
    "    callbacks = []\n",
    "    if output_dir:\n",
    "        tensorboard_callback = TensorBoard(log_dir=output_dir)\n",
    "        callbacks = [tensorboard_callback]\n",
    "\n",
    "    history = model.fit(\n",
    "        train_data,\n",
    "        validation_data=validation_data,\n",
    "        epochs=num_epochs,\n",
    "        steps_per_epoch=steps_per_epoch,\n",
    "        verbose=2,\n",
    "        callbacks=callbacks)\n",
    "\n",
    "    if output_dir:\n",
    "        export_path = os.path.join(output_dir, 'keras_export')\n",
    "        model.save(export_path, save_format='tf')\n",
    "\n",
    "    return history\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local Training\n",
    "\n",
    "With everything set up, let's run locally to test the code. Some of the previous tests have been copied over into a testing script `mnist_models/trainer/test.py` to make sure the model still passes our previous checks. On `line 13`, you can specify which model types you would like to check. `line 14` and `line 15` has the number of epochs and steps per epoch respectively.\n",
    "\n",
    "Moment of truth! Run the code below to check your models against the unit tests. If you see \"OK\" at the end when it's finished running, congrats! You've passed the tests!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-01-16 05:18:44.945076: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\n",
      "2020-01-16 05:18:48.133198: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\n",
      "2020-01-16 05:18:48.136960: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.137440: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \n",
      "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.562\n",
      "pciBusID: 0000:00:04.0\n",
      "2020-01-16 05:18:48.137493: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\n",
      "2020-01-16 05:18:48.139017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0\n",
      "2020-01-16 05:18:48.140478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0\n",
      "2020-01-16 05:18:48.140937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0\n",
      "2020-01-16 05:18:48.143407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0\n",
      "2020-01-16 05:18:48.145015: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0\n",
      "2020-01-16 05:18:48.149588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
      "2020-01-16 05:18:48.149825: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.150331: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.150660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\n",
      "2020-01-16 05:18:48.159213: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz\n",
      "2020-01-16 05:18:48.159592: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55adaacb8770 executing computations on platform Host. Devices:\n",
      "2020-01-16 05:18:48.159638: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version\n",
      "2020-01-16 05:18:48.225613: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.226166: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55adaad2b9b0 executing computations on platform CUDA. Devices:\n",
      "2020-01-16 05:18:48.226209: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
      "2020-01-16 05:18:48.226552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.227025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \n",
      "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.562\n",
      "pciBusID: 0000:00:04.0\n",
      "2020-01-16 05:18:48.227099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\n",
      "2020-01-16 05:18:48.227161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0\n",
      "2020-01-16 05:18:48.227219: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0\n",
      "2020-01-16 05:18:48.227255: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0\n",
      "2020-01-16 05:18:48.227287: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0\n",
      "2020-01-16 05:18:48.227319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0\n",
      "2020-01-16 05:18:48.227352: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
      "2020-01-16 05:18:48.227442: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.227963: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.228406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\n",
      "2020-01-16 05:18:48.228473: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\n",
      "2020-01-16 05:18:48.754118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2020-01-16 05:18:48.754187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 \n",
      "2020-01-16 05:18:48.754204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N \n",
      "2020-01-16 05:18:48.754741: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.755241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:18:48.755690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10285 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "..\n",
      "*** Building model for linear ***\n",
      "\n",
      "Train for 100 steps, validate for 100 steps\n",
      "Epoch 1/10\n",
      "2020-01-16 05:19:00.979496: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0\n",
      "100/100 - 7s - loss: 1.3553 - accuracy: 0.6579 - val_loss: 0.8047 - val_accuracy: 0.8377\n",
      "Epoch 2/10\n",
      "100/100 - 1s - loss: 0.6943 - accuracy: 0.8436 - val_loss: 0.5564 - val_accuracy: 0.8715\n",
      "Epoch 3/10\n",
      "100/100 - 1s - loss: 0.5415 - accuracy: 0.8682 - val_loss: 0.4654 - val_accuracy: 0.8898\n",
      "Epoch 4/10\n",
      "100/100 - 1s - loss: 0.4609 - accuracy: 0.8803 - val_loss: 0.4194 - val_accuracy: 0.8953\n",
      "Epoch 5/10\n",
      "100/100 - 1s - loss: 0.3979 - accuracy: 0.8985 - val_loss: 0.3917 - val_accuracy: 0.9005\n",
      "Epoch 6/10\n",
      "100/100 - 2s - loss: 0.4024 - accuracy: 0.8901 - val_loss: 0.3669 - val_accuracy: 0.9056\n",
      "Epoch 7/10\n",
      "100/100 - 7s - loss: 0.3874 - accuracy: 0.8961 - val_loss: 0.3541 - val_accuracy: 0.9063\n",
      "Epoch 8/10\n",
      "100/100 - 1s - loss: 0.3674 - accuracy: 0.9021 - val_loss: 0.3420 - val_accuracy: 0.9093\n",
      "Epoch 9/10\n",
      "100/100 - 1s - loss: 0.3477 - accuracy: 0.9090 - val_loss: 0.3360 - val_accuracy: 0.9112\n",
      "Epoch 10/10\n",
      "100/100 - 1s - loss: 0.3263 - accuracy: 0.9106 - val_loss: 0.3241 - val_accuracy: 0.9134\n",
      "\n",
      "*** Building model for dnn ***\n",
      "\n",
      "100/100 - 7s - loss: 0.5730 - accuracy: 0.8386 - val_loss: 0.3121 - val_accuracy: 0.9089\n",
      "Epoch 2/10\n",
      "100/100 - 1s - loss: 0.2515 - accuracy: 0.9274 - val_loss: 0.2085 - val_accuracy: 0.9395\n",
      "Epoch 3/10\n",
      "100/100 - 1s - loss: 0.2000 - accuracy: 0.9351 - val_loss: 0.1880 - val_accuracy: 0.9434\n",
      "Epoch 4/10\n",
      "100/100 - 1s - loss: 0.1796 - accuracy: 0.9469 - val_loss: 0.1460 - val_accuracy: 0.9574\n",
      "Epoch 5/10\n",
      "100/100 - 1s - loss: 0.1520 - accuracy: 0.9531 - val_loss: 0.1406 - val_accuracy: 0.9557\n",
      "Epoch 6/10\n",
      "100/100 - 1s - loss: 0.1351 - accuracy: 0.9581 - val_loss: 0.1177 - val_accuracy: 0.9643\n",
      "Epoch 7/10\n",
      "100/100 - 6s - loss: 0.1039 - accuracy: 0.9686 - val_loss: 0.1038 - val_accuracy: 0.9685\n",
      "Epoch 8/10\n",
      "100/100 - 1s - loss: 0.0944 - accuracy: 0.9725 - val_loss: 0.1103 - val_accuracy: 0.9650\n",
      "Epoch 9/10\n",
      "100/100 - 1s - loss: 0.1110 - accuracy: 0.9663 - val_loss: 0.1052 - val_accuracy: 0.9678\n",
      "Epoch 10/10\n",
      "100/100 - 1s - loss: 0.0906 - accuracy: 0.9735 - val_loss: 0.0941 - val_accuracy: 0.9708\n",
      "\n",
      "*** Building model for dnn_dropout ***\n",
      "\n",
      "Train for 100 steps, validate for 100 steps\n",
      "Epoch 1/10\n",
      "100/100 - 7s - loss: 0.6572 - accuracy: 0.8011 - val_loss: 0.2871 - val_accuracy: 0.9177\n",
      "Epoch 2/10\n",
      "100/100 - 1s - loss: 0.3085 - accuracy: 0.9079 - val_loss: 0.2138 - val_accuracy: 0.9338\n",
      "Epoch 3/10\n",
      "100/100 - 1s - loss: 0.2339 - accuracy: 0.9317 - val_loss: 0.1914 - val_accuracy: 0.9407\n",
      "Epoch 4/10\n",
      "100/100 - 1s - loss: 0.1930 - accuracy: 0.9415 - val_loss: 0.1504 - val_accuracy: 0.9537\n",
      "Epoch 5/10\n",
      "100/100 - 1s - loss: 0.1867 - accuracy: 0.9411 - val_loss: 0.1413 - val_accuracy: 0.9566\n",
      "Epoch 6/10\n",
      "100/100 - 1s - loss: 0.1594 - accuracy: 0.9540 - val_loss: 0.1248 - val_accuracy: 0.9603\n",
      "Epoch 7/10\n",
      "100/100 - 6s - loss: 0.1285 - accuracy: 0.9611 - val_loss: 0.1075 - val_accuracy: 0.9668\n",
      "Epoch 8/10\n",
      "100/100 - 1s - loss: 0.1308 - accuracy: 0.9602 - val_loss: 0.1077 - val_accuracy: 0.9671\n",
      "Epoch 9/10\n",
      "100/100 - 1s - loss: 0.1215 - accuracy: 0.9643 - val_loss: 0.1063 - val_accuracy: 0.9667\n",
      "Epoch 10/10\n",
      "100/100 - 1s - loss: 0.1192 - accuracy: 0.9646 - val_loss: 0.1011 - val_accuracy: 0.9685\n",
      "\n",
      "*** Building model for cnn ***\n",
      "\n",
      "Train for 100 steps, validate for 100 steps\n",
      "Epoch 1/10\n",
      "100/100 - 9s - loss: 0.6915 - accuracy: 0.7779 - val_loss: 0.1829 - val_accuracy: 0.9478\n",
      "Epoch 2/10\n",
      "100/100 - 2s - loss: 0.1862 - accuracy: 0.9447 - val_loss: 0.1235 - val_accuracy: 0.9612\n",
      "Epoch 3/10\n",
      "100/100 - 2s - loss: 0.1354 - accuracy: 0.9576 - val_loss: 0.0830 - val_accuracy: 0.9735\n",
      "Epoch 4/10\n",
      "100/100 - 2s - loss: 0.1206 - accuracy: 0.9615 - val_loss: 0.0649 - val_accuracy: 0.9812\n",
      "Epoch 5/10\n",
      "100/100 - 2s - loss: 0.0916 - accuracy: 0.9718 - val_loss: 0.0534 - val_accuracy: 0.9829\n",
      "Epoch 6/10\n",
      "100/100 - 2s - loss: 0.1032 - accuracy: 0.9684 - val_loss: 0.0607 - val_accuracy: 0.9811\n",
      "Epoch 7/10\n",
      "100/100 - 7s - loss: 0.0644 - accuracy: 0.9811 - val_loss: 0.0542 - val_accuracy: 0.9835\n",
      "Epoch 8/10\n",
      "100/100 - 2s - loss: 0.0692 - accuracy: 0.9791 - val_loss: 0.0483 - val_accuracy: 0.9839\n",
      "Epoch 9/10\n",
      "100/100 - 2s - loss: 0.0644 - accuracy: 0.9809 - val_loss: 0.0451 - val_accuracy: 0.9857\n",
      "Epoch 10/10\n",
      "100/100 - 2s - loss: 0.0673 - accuracy: 0.9790 - val_loss: 0.0432 - val_accuracy: 0.9853\n",
      "...\n",
      "----------------------------------------------------------------------\n",
      "Ran 5 tests in 132.269s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "!python3 -m mnist_models.trainer.test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know that our models are working as expected, let's run it on the [Google Cloud AI Platform](https://cloud.google.com/ml-engine/docs/). We can run it as a python module locally first using the command line.\n",
    "\n",
    "The below cell transfers some of our variables to the command line as well as create a job directory including a timestamp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_time = datetime.now().strftime(\"%y%m%d_%H%M%S\")\n",
    "model_type = 'cnn' # \"linear\", \"cnn\", \"dnn_dropout\", or \"dnn\"\n",
    "\n",
    "os.environ[\"MODEL_TYPE\"] = model_type\n",
    "os.environ[\"JOB_DIR\"] = \"mnist_models/models/{}_{}/\".format(\n",
    "    model_type, current_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below runs the local version of the code. The epochs and steps_per_epoch flag can be changed to run for longer or shorter, as defined in our `mnist_models/trainer/task.py` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 50 steps, validate for 100 steps\n",
      "Epoch 1/5\n",
      "50/50 - 10s - loss: 1.0179 - accuracy: 0.6738 - val_loss: 0.4321 - val_accuracy: 0.8669\n",
      "Epoch 2/5\n",
      "50/50 - 2s - loss: 0.3472 - accuracy: 0.8968 - val_loss: 0.1736 - val_accuracy: 0.9497\n",
      "Epoch 3/5\n",
      "50/50 - 2s - loss: 0.2108 - accuracy: 0.9368 - val_loss: 0.1329 - val_accuracy: 0.9587\n",
      "Epoch 4/5\n",
      "50/50 - 2s - loss: 0.1708 - accuracy: 0.9464 - val_loss: 0.1116 - val_accuracy: 0.9641\n",
      "Epoch 5/5\n",
      "50/50 - 2s - loss: 0.1560 - accuracy: 0.9514 - val_loss: 0.0938 - val_accuracy: 0.9726\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2020-01-16 05:25:13.487462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\n",
      "2020-01-16 05:25:15.782968: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\n",
      "2020-01-16 05:25:15.786337: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:15.786747: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \n",
      "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.562\n",
      "pciBusID: 0000:00:04.0\n",
      "2020-01-16 05:25:15.786787: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\n",
      "2020-01-16 05:25:15.788375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0\n",
      "2020-01-16 05:25:15.789817: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0\n",
      "2020-01-16 05:25:15.790163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0\n",
      "2020-01-16 05:25:15.792126: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0\n",
      "2020-01-16 05:25:15.793596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0\n",
      "2020-01-16 05:25:15.798378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
      "2020-01-16 05:25:15.798647: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:15.799173: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:15.799500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\n",
      "2020-01-16 05:25:15.811621: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz\n",
      "2020-01-16 05:25:15.811990: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558dad09ee00 executing computations on platform Host. Devices:\n",
      "2020-01-16 05:25:15.812024: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version\n",
      "2020-01-16 05:25:15.879488: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:15.879961: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x558dad1120a0 executing computations on platform CUDA. Devices:\n",
      "2020-01-16 05:25:15.879997: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
      "2020-01-16 05:25:15.880233: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:15.880554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: \n",
      "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.562\n",
      "pciBusID: 0000:00:04.0\n",
      "2020-01-16 05:25:15.880604: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\n",
      "2020-01-16 05:25:15.880640: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0\n",
      "2020-01-16 05:25:15.880666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0\n",
      "2020-01-16 05:25:15.880689: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0\n",
      "2020-01-16 05:25:15.880711: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0\n",
      "2020-01-16 05:25:15.880727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0\n",
      "2020-01-16 05:25:15.880750: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
      "2020-01-16 05:25:15.880818: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:15.881232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:15.881623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0\n",
      "2020-01-16 05:25:15.881705: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0\n",
      "2020-01-16 05:25:16.398516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2020-01-16 05:25:16.398584: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 \n",
      "2020-01-16 05:25:16.398596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N \n",
      "2020-01-16 05:25:16.399117: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:16.399539: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2020-01-16 05:25:16.399897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10285 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
      "2020-01-16 05:25:18.352074: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0\n",
      "2020-01-16 05:25:23.815108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
      "2020-01-16 05:25:24.920631: I tensorflow/core/profiler/lib/profiler_session.cc:184] Profiler session started.\n",
      "2020-01-16 05:25:24.922583: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcupti.so.10.0\n",
      "2020-01-16 05:25:25.180703: I tensorflow/core/platform/default/device_tracer.cc:588] Collecting 120 kernel records, 8 memcpy records.\n",
      "WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.128515). Check your callbacks.\n",
      "2020-01-16 05:25:34.101811: W tensorflow/python/util/util.cc:299] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1781: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python3 -m mnist_models.trainer.task \\\n",
    "    --job-dir=$JOB_DIR \\\n",
    "    --epochs=5 \\\n",
    "    --steps_per_epoch=50 \\\n",
    "    --model_type=$MODEL_TYPE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training on the cloud\n",
    "\n",
    "Since we're using an unreleased version of TensorFlow on AI Platform, we can instead use a [Deep Learning Container](https://cloud.google.com/ai-platform/deep-learning-containers/docs/overview) in order to take advantage of libraries and applications not normally packaged with AI Platform. Below is a simple [Dockerlife](https://docs.docker.com/engine/reference/builder/) which copies our code to be used in a TF2 environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mnist_models/Dockerfile\n"
     ]
    }
   ],
   "source": [
    "%%writefile mnist_models/Dockerfile\n",
    "FROM gcr.io/deeplearning-platform-release/tf2-cpu\n",
    "COPY mnist_models/trainer /mnist_models/trainer\n",
    "ENTRYPOINT [\"python3\", \"-m\", \"mnist_models.trainer.task\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below command builds the image and ships it off to Google Cloud so it can be used for AI Platform. When built, it will show up [here](http://console.cloud.google.com/gcr) with the name `mnist_models`. ([Click here](https://console.cloud.google.com/cloud-build) to enable Cloud Build)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending build context to Docker daemon  6.766MB\n",
      "Step 1/3 : FROM gcr.io/deeplearning-platform-release/tf2-cpu\n",
      " ---> e493f17c90d0\n",
      "Step 2/3 : COPY mnist_models/trainer /mnist_models/trainer\n",
      " ---> 9c7bb60ef956\n",
      "Step 3/3 : ENTRYPOINT [\"python3\", \"-m\", \"mnist_models.trainer.task\"]\n",
      " ---> Running in 3b7135292970\n",
      "Removing intermediate container 3b7135292970\n",
      " ---> 6270b4938691\n",
      "Successfully built 6270b4938691\n",
      "Successfully tagged gcr.io/ddetering-experimental/mnist_models:latest\n"
     ]
    }
   ],
   "source": [
    "!docker build -f mnist_models/Dockerfile -t $IMAGE_URI ./"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The push refers to repository [gcr.io/ddetering-experimental/mnist_models]\n",
      "\n",
      "\u001b[1B3439f44a: Preparing \n",
      "\u001b[1B581319d1: Preparing \n",
      "\u001b[1Bd92bb97d: Preparing \n",
      "\u001b[1B218cebaf: Preparing \n",
      "\u001b[1B750480c5: Preparing \n",
      "\u001b[1B1a04733b: Preparing \n",
      "\u001b[1B67e0ba2d: Preparing \n",
      "\u001b[1B50747c65: Preparing \n",
      "\u001b[1B36f9bda1: Preparing \n",
      "\u001b[1B7efe6784: Preparing \n",
      "\u001b[1B27b0e386: Preparing \n",
      "\u001b[1B4689ea9a: Preparing \n",
      "\u001b[1B971c1728: Preparing \n",
      "\u001b[1B63e1ea3e: Preparing \n",
      "\u001b[1B2b9c88ab: Preparing \n",
      "\u001b[1Bdb0181d2: Preparing \n",
      "\u001b[1B197acff6: Preparing \n",
      "\u001b[1B87810c15: Preparing \n",
      "\u001b[1Be11ab4a2: Preparing \n",
      "\u001b[1B13bce073: Preparing \n",
      "\u001b[1Be43028b3: Preparing \n",
      "\u001b[22B439f44a: Pushed lready exists 2kB\u001b[18A\u001b[2K\u001b[16A\u001b[2K\u001b[12A\u001b[2K\u001b[9A\u001b[2K\u001b[7A\u001b[2K\u001b[6A\u001b[2K\u001b[2A\u001b[2K\u001b[1A\u001b[2K\u001b[22A\u001b[2Klatest: digest: sha256:9ff5bfb9f1ae5a3d58063b2e8165e855b5c770549aac317b19d344b9ec336fdc size: 4925\n"
     ]
    }
   ],
   "source": [
    "!docker push $IMAGE_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can kickoff the [AI Platform training job](https://cloud.google.com/sdk/gcloud/reference/ai-platform/jobs/submit/training). We can pass in our docker image using the `master-image-uri` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_time = datetime.now().strftime(\"%y%m%d_%H%M%S\")\n",
    "model_type = 'cnn' # \"linear\", \"cnn\", \"dnn_dropout\", or \"dnn\"\n",
    "\n",
    "os.environ[\"MODEL_TYPE\"] = model_type\n",
    "os.environ[\"JOB_DIR\"] = \"gs://{}/mnist_{}_{}/\".format(\n",
    "    BUCKET, model_type, current_time)\n",
    "os.environ[\"JOB_NAME\"] = \"mnist_{}_{}\".format(\n",
    "    model_type, current_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AI platform job could take around 10 minutes to complete. Enable the **AI Platform Training & Prediction API**, if required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://ddetering-experimental/mnist_cnn_200116_053228/ us-central1 mnist_cnn_200116_053228\n",
      "jobId: mnist_cnn_200116_053228\n",
      "state: QUEUED\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Job [mnist_cnn_200116_053228] submitted successfully.\n",
      "Your job is still active. You may view the status of your job with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs describe mnist_cnn_200116_053228\n",
      "\n",
      "or continue streaming the logs with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs stream-logs mnist_cnn_200116_053228\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo $JOB_DIR $REGION $JOB_NAME\n",
    "gcloud ai-platform jobs submit training $JOB_NAME \\\n",
    "    --staging-bucket=gs://$BUCKET \\\n",
    "    --region=$REGION \\\n",
    "    --master-image-uri=$IMAGE_URI \\\n",
    "    --scale-tier=BASIC_GPU \\\n",
    "    --job-dir=$JOB_DIR \\\n",
    "    -- \\\n",
    "    --model_type=$MODEL_TYPE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can't wait to see the results? Run the code below and copy the output into the [Google Cloud Shell](https://console.cloud.google.com/home/dashboard?cloudshell=true) to follow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying and predicting with model\n",
    "\n",
    "Once you have a model you're proud of, let's deploy it! All we need to do is give AI Platform the location of the model. Below uses the keras export path of the previous job, but `${JOB_DIR}keras_export/` can always be changed to a different path.\n",
    "\n",
    "Uncomment the delete commands below if you are getting an \"already exists error\" and want to deploy a new model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
       "Updated property [ai_platform/region].\n"
      ]
     }
   ],
   "source": [
    "%%bash\n",
    "gcloud config set ai_platform/region global\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting and deploying mnist cnn from gs://ddetering-experimental/mnist_cnn_200116_053228/keras_export/ ... this will take a few minutes\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created ml engine model [projects/ddetering-experimental/models/mnist].\n",
      "Creating version (this might take a few minutes)......\n",
      ".......................................................................................................................................................................................................................................................................................................................................................................................done.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "MODEL_NAME=\"mnist\"\n",
    "MODEL_VERSION=${MODEL_TYPE}\n",
    "MODEL_LOCATION=${JOB_DIR}keras_export/\n",
    "echo \"Deleting and deploying $MODEL_NAME $MODEL_VERSION from $MODEL_LOCATION ... this will take a few minutes\"\n",
    "#yes | gcloud ai-platform versions delete ${MODEL_VERSION} --model ${MODEL_NAME}\n",
    "#yes | gcloud ai-platform models delete ${MODEL_NAME}\n",
    "gcloud ai-platform models create ${MODEL_NAME} --regions $REGION\n",
    "gcloud ai-platform versions create ${MODEL_VERSION} \\\n",
    "    --model ${MODEL_NAME} \\\n",
    "    --origin ${MODEL_LOCATION} \\\n",
    "    --framework tensorflow \\\n",
    "    --runtime-version=2.6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To predict with the model, let's take one of the example images.\n",
    "\n",
    "**TODO 4**: Write a `.json` file with image data to send to an AI Platform deployed model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADg5JREFUeJzt3XuMXPV5xvHnwawNmEtsaDYuuDFNKKlDywIb0whaSEgiYiUFqhZhqanT0jhSAyoVSYNADfxRKahtEkhLUU1wYyIuScrFboVSqGuJRCEuCzg2xlButrBlbBLT2EnA2N63f+xxtIGd36zndmZ5vx9pNTPnPWfOqyM/PjPzOzM/R4QA5HNI3Q0AqAfhB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+Q1KG93Nl0z4jDNLOXuwRSeU0/0+uxx5NZt63w2z5f0o2Spkn6WkRcX1r/MM3UmT6vnV0CKFgTqya9bssv+21Pk3STpI9Kmi9pke35rT4fgN5q5z3/AknPRsTzEfG6pLskXdCZtgB0WzvhP17Si+Meb6mW/RLbS2yP2B7Zqz1t7A5AJ3X90/6IWBoRwxExPKAZ3d4dgElqJ/xbJc0d9/iEahmAKaCd8D8i6STbJ9qeLukSSSs70xaAbmt5qC8i9tm+TNJ/amyob1lEbOhYZwC6qq1x/oi4X9L9HeoFQA9xeS+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJtTVLr+1NknZL2i9pX0QMd6IpAN3XVvgrH4iIH3XgeQD0EC/7gaTaDX9IesD2o7aXdKIhAL3R7sv+syNiq+23S3rQ9lMR8dD4Far/FJZI0mE6os3dAeiUts78EbG1ut0h6V5JCyZYZ2lEDEfE8IBmtLM7AB3Ucvhtz7R91IH7kj4i6YlONQagu9p52T8o6V7bB57njoj4Tke6AtB1LYc/Ip6XdGoHewHQQwz1AUkRfiApwg8kRfiBpAg/kBThB5LqxLf60Mf2n3t6sX7oF7YX6/9+8spifcDTivW9sb9h7ay1lxS3PfaagWLdm7YW6z/++PyGtdn3la9HG929u1h/K+DMDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJMc4/BXhG+ReQdv/+UMPatV9cVtz2nMN/XqyPFqvS3ijXRwvP8N2hO4rbnv43nyzWT31H+dy1Yt4/Nay9722XF7cd/MfvF+tvBZz5gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApxvmngD3n/lax/t83NB7Pbmb1q0cW61/42z8r1gd+3mSgv2DXO8vnnunlSxD0158tX8Pwk9F9DWtHbmv8OwNZcOYHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaSajvPbXibpY5J2RMQp1bLZkr4paZ6kTZIujohXutfmW1u8vzzT+Rdv/peWn3vRcwuL9V3Xzi3WZ61+uOV9N3PMu08s1oe+/Vyx/pvTy+eu96z4q4a13/i3NcVtM5jMmf/rks5/w7KrJK2KiJMkraoeA5hCmoY/Ih6StPMNiy+QtLy6v1zShR3uC0CXtfqefzAitlX3X5I02KF+APRI2x/4RURIaniBt+0ltkdsj+zVnnZ3B6BDWg3/dttzJKm63dFoxYhYGhHDETE8oPIPUQLonVbDv1LS4ur+YkkrOtMOgF5pGn7bd0p6WNLJtrfYvlTS9ZI+bPsZSR+qHgOYQpqO80fEogal8zrcS1qvXPNqsX5Gk3dLC5/6g4a1aZ89urjttMcfKz95F/3fGeXPia99+7faev65D7S1+VseV/gBSRF+ICnCDyRF+IGkCD+QFOEHkuKnu3vghbt+u1jfcNq/Futb9pWHAg+5ZlbDWjy+rrhtt5WmF3/3FU8Wtz2kybnpTzeXR5sPv+9/ivXsOPMDSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKM8/fAn8wvjzeParRY37yv/LVc/aC+sfzSOL4kPX1D458lX/FrNxW3LR8VafPfn1ysHyF+nruEMz+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJMU4P4qmvbc8lr7x8mOK9ac+Xh7LL1n96pHF+lHff6FY39/ynnPgzA8kRfiBpAg/kBThB5Ii/EBShB9IivADSTUd57e9TNLHJO2IiFOqZddJ+pSkl6vVro6I+7vV5FR39wtDxfrnjl1frJ8242fF+u+ue+2ge5qsBUfcU6x/4PDyvpt9J7/kyh/+YbF+wvYNbTw7JnPm/7qk8ydY/pWIGKr+CD4wxTQNf0Q8JGlnD3oB0EPtvOe/zPY628tsN54vCkBfajX8N0t6l6QhSdskfanRiraX2B6xPbJXe1rcHYBOayn8EbE9IvZHxKikWyQtKKy7NCKGI2J4QOUfewTQOy2F3/accQ8vkvREZ9oB0CuTGeq7U9K5ko6zvUXStZLOtT0kKSRtkvTpLvYIoAscET3b2dGeHWe6PKf6W9EhRx1VrI/eV/5O/H+8Z0V5+7ZG09tzzucvL9ZHF/24Ye27Q3cUtz3/0r8o1qd/55FiPaM1sUq7Yqcnsy5X+AFJEX4gKcIPJEX4gaQIP5AU4QeS4qe7e2B09+7yCueV6x+8qDzkteOM1v8Pn7WxPNR7zO0/KNZf/kb5ku2nhu5qWLv1J/OK2x6xYVuxvq9YRTOc+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcb5p4Aj7l1TrM+7t0eNTOCpD36tWC993fimp88pbvurLz7ZUk+YHM78QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4/womvbek5us8Wixunnf6w1rg189rIWO0Cmc+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gqabj/LbnSrpN0qCkkLQ0Im60PVvSNyXNk7RJ0sUR8Ur3WkUdnr92elvb/9Hjf96w9o7Vj7X13GjPZM78+yRdGRHzJf2OpM/Yni/pKkmrIuIkSauqxwCmiKbhj4htEfFYdX+3pI2Sjpd0gaTl1WrLJV3YrSYBdN5Bvee3PU/SaZLWSBqMiAPzKb2ksbcFAKaISYff9pGS7pZ0RUTsGl+LiNDY5wETbbfE9ojtkb0qz+sGoHcmFX7bAxoL/u0RcU+1eLvtOVV9jqQdE20bEUsjYjgihgc0oxM9A+iApuG3bUm3StoYEV8eV1opaXF1f7GkFZ1vD0C3TOYrvWdJ+oSk9bbXVsuulnS9pG/ZvlTSZkkXd6dFdFO8/9RifeWZ/9zkGcpfy/WqWQfZEXqlafgj4nuS3KB8XmfbAdArXOEHJEX4gaQIP5AU4QeSIvxAUoQfSIqf7k5ux/tmFusnHloexy9NwS1Jh7424VXf6AOc+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcb5k3vtuPI4fLNx/Bt2zi/Wj73l4YPuCb3BmR9IivADSRF+ICnCDyRF+IGkCD+QFOEHkmKcP7k/vnB1W9svW/GhYn2eGOfvV5z5gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiCppuP8tudKuk3SoKSQtDQibrR9naRPSXq5WvXqiLi/W42iO+5+YahY/9yx63vUCXptMhf57JN0ZUQ8ZvsoSY/afrCqfSUi/qF77QHolqbhj4htkrZV93fb3ijp+G43BqC7Duo9v+15kk6TtKZadJntdbaX2Z7VYJsltkdsj+zVnraaBdA5kw6/7SMl3S3piojYJelmSe+SNKSxVwZfmmi7iFgaEcMRMTygGR1oGUAnTCr8tgc0FvzbI+IeSYqI7RGxPyJGJd0iaUH32gTQaU3Db9uSbpW0MSK+PG75nHGrXSTpic63B6BbJvNp/1mSPiFpve211bKrJS2yPaSx4b9Nkj7dlQ7RVbFqdrF+9QlnFuuDI/s72Q56aDKf9n9PkicoMaYPTGFc4QckRfiBpAg/kBThB5Ii/EBShB9IyhHlKZo76WjPjjN9Xs/2B2SzJlZpV+ycaGj+TTjzA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBSPR3nt/2ypM3jFh0n6Uc9a+Dg9Gtv/dqXRG+t6mRv74yIX5nMij0N/5t2bo9ExHBtDRT0a2/92pdEb62qqzde9gNJEX4gqbrDv7Tm/Zf0a2/92pdEb62qpbda3/MDqE/dZ34ANakl/LbPt/207WdtX1VHD43Y3mR7ve21tkdq7mWZ7R22nxi3bLbtB20/U91OOE1aTb1dZ3trdezW2l5YU29zba+2/aTtDbb/slpe67Er9FXLcev5y37b0yT9r6QPS9oi6RFJiyLiyZ420oDtTZKGI6L2MWHbvyfpp5Jui4hTqmV/J2lnRFxf/cc5KyI+3ye9XSfpp3XP3FxNKDNn/MzSki6U9EnVeOwKfV2sGo5bHWf+BZKejYjnI+J1SXdJuqCGPvpeRDwkaecbFl8gaXl1f7nG/vH0XIPe+kJEbIuIx6r7uyUdmFm61mNX6KsWdYT/eEkvjnu8Rf015XdIesD2o7aX1N3MBAaradMl6SVJg3U2M4GmMzf30htmlu6bY9fKjNedxgd+b3Z2RJwu6aOSPlO9vO1LMfaerZ+GayY1c3OvTDCz9C/UeexanfG60+oI/1ZJc8c9PqFa1hciYmt1u0PSveq/2Ye3H5gktbrdUXM/v9BPMzdPNLO0+uDY9dOM13WE/xFJJ9k+0fZ0SZdIWllDH29ie2b1QYxsz5T0EfXf7MMrJS2u7i+WtKLGXn5Jv8zc3GhmadV87PpuxuuI6PmfpIUa+8T/OUnX1NFDg75+XdIPq78Ndfcm6U6NvQzcq7HPRi6VdKykVZKekfRfkmb3UW/fkLRe0jqNBW1OTb2drbGX9Oskra3+FtZ97Ap91XLcuMIPSIoP/ICkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJPX/rJw9J1q+cE8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import json, codecs\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "from mnist_models.trainer import util\n",
    "\n",
    "HEIGHT = 28\n",
    "WIDTH = 28\n",
    "IMGNO = 12\n",
    "\n",
    "mnist = tf.keras.datasets.mnist.load_data()\n",
    "(x_train, y_train), (x_test, y_test) = mnist\n",
    "test_image = x_test[IMGNO]\n",
    "\n",
    "jsondata = test_image.reshape(HEIGHT, WIDTH, 1).tolist()\n",
    "json.dump(jsondata, codecs.open(\"test.json\", \"w\", encoding = \"utf-8\"))\n",
    "plt.imshow(test_image.reshape(HEIGHT, WIDTH));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can send it to the prediction service. The output will have a 1 in the index of the corresponding digit it is predicting. Congrats! You've completed the lab!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SOFTMAX_3\n",
      "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "gcloud ai-platform predict \\\n",
    "    --model=mnist \\\n",
    "    --version=${MODEL_TYPE} \\\n",
    "    --json-instances=./test.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2022 Google Inc.\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at\n",
    "http://www.apache.org/licenses/LICENSE-2.0\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
